[3.15] gh-151613: Fix remote debugging frame cache ABA - #152448
Conversation
The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack. This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches. (cherry picked from commit 8cda6ae)
|
I'm sorry but is it intentional that this change breaks extension ABI? If yes, could we please have a warning about such things in release notes? It's not nice when we keep being asked to test things early, then learn the hard way that suddenly everything using |
|
The ABI is only frozen after the first release candidate as per our policy. We don't keep track of ABI breakage changes before we freeze ABI but we can discuss doing it, but as things stand today the only guarantees start after RC1 |
|
Sure, just want to make sure that the change is intentional before we tell hundreds of people who already started testing that they have to rebuild everything, and hope nothing goes out of order so things don't start segfaulting randomly. |
The change is intentional in the sense that was required for the fix. As you can imagine we don't want to break people unnecessarily even if we are still allowed to do so. |
|
I think tracking extension module ABI breakages in beta versions (and rc1) would be helpful. I'll start a discussion on dpo after I recover from EuroPython. |
|
To avoid ABI breakage in the future, we should move PyInterpreterState structure to the internal C API! But that's a lot of work :-( Currently, multiple 3rd party C extensions access it directly, instead of calling getter or setter functions. |
This reverts commit 885a7ca6. The library is not what failed: on 3.15.0b4 the whole suite passes, 77 tests, once cffi is built from source. What failed is cffi's own cp315 wheel, compiled against 3.15b3, whose PyThreadState grew a field in b4 (python/cpython#152448). The offset it was compiled with is now wrong, so the process segfaults in take_gil the moment C re-enters Python through a libffi closure — here the illegal_callback that secp256k1_ec_pubkey_parse invokes on invalid input. Upstream is python-cffi/cffi#263, whose fix is a new cffi release. Keeping the jobs would mean thirteen red ones in every run until 3.15 reaches release candidate, for a cause outside this repository, and a permanently red matrix is one nobody reads. Building cffi from source in CI would hide the red, at the price of a workaround nobody would remember to remove. So 3.15 comes back when there is something to learn from it: cffi wheels that match the interpreter, or the release candidate, which is also when cibuildwheel starts producing a cp315 wheel without a flag and test-static can take 3.15 in the same move. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 participants
Backport of gh-151613 to Python 3.15.
Original commit: 8cda6ae
Original PR: GH-151614